Watermarking scheme evaluation tool

Digital watermarking has been presented as a solution for protection against illegal copying of multimedia objects and dozens algorithms have been proposed. Some problems seriously darken the future of this technology though. One of them is that the requirements, tools and methodologies to assess the current technologies are almost inexistent. The lack of benchmarking of current algorithms is blatant. This confuses rights holders as well as software and hardware manufacturers and prevents them from using the solution appropriate to their needs. Indeed basing long-lived protection schemes on badly tested watermarking technology does not make sense. In this paper we will present the architecture of an evaluation tool being developed. 1. Need for evaluation Digital watermarking remains a largely untested field and only very few large industrial consortiums have published requirements against which watermarking algorithms should be tested. For instance the International Federation for the Phonographic Industry led one of the first large scale comparative testing of watermarking algorithm for audio [1]. In general, a number of broad claims have been made about the ‘robustness’ of various digital watermarking or fingerprinting methods but the growing number of attacks against such systems (e.g., [2, 3]) has shown that far more research is actually required to improve the quality of existing watermarking methods. With a common benchmark authors and watermarking software providers would just need to provide a table of results, which would give a good and reliable summary of the performances of the proposed scheme [4]. So end users can check whether their basic requirements are satisfied, researchers can compare different algorithms and see how a method can be improved or whether a newly added feature actually improves the reliability of the whole method and the industry can properly evaluate risks associated to the use of a particular solution by knowing which level of reliability can be achieved by each contender. 2. Evaluation tool As a first step towards a widely accepted way to evaluate watermarking schemes we started to implement an automated benchmark server for digital watermarking schemes. The idea is to allow users to send a binary library of their scheme to the server which in turns runs a series of tests on this library and keeps the results in a database accessible to the scheme owner or to all ‘watermarkers’ through the Web. Simplicity—In order to be widely accepted this service has a simple interface with existing watermarking libraries (only three functions must be provided). It also takes into account the application of the watermarking scheme by proposing different evaluation profiles (sets of tests and images) and strengths. These goals a reflected in Figure 1 and will be detailed in the next sections. Customisation—For each type of watermarking scheme, we want to use a different evaluation profile without having to recompile the application tool. Definition of the profiles is not an easy task and requires agreement among the watermarking community. As we will see however, the choice of these profiles does not affect the design of the evaluation service and can be done later and tuned after experimenting the service. Modularity and choice of tests—Watermarking algorithms are often used in larger system designed to achieve certain goals (e.g., prevention of illegal copying, trading of images). But here we are only concerned with the evaluation of watermarking (so the signal processing aspects) within the larger system not the effectiveness of the full system to achieve its goals. So the main functionalities we wish to evaluate include the perceptibility of the scheme, its capacity, its reliability (robustness to attacks and false alarm rate) and its performances (mainly the speed of execution). For each of these set of tests we have implemented ad-hoc libraries which are built easily on top of the core libraries as shown in the next section.  Perceptibility characterises the amount of distortion introduced during by the watermarking scheme itself. The problem here is very similar to the evaluation of compression algorithms. We allow the addition and use of different quality metrics, the simplest and most widely used one being the P.S.N.R.  The capacity of a scheme is the amount of information one can hide. In most applications the capacity will be a fixed constraint of the system so robustness test will be done with a random payload of given size. However knowing the relation between capacity and robustness is very important and our benchmark provide a test that help to analyse this trade-off by drawing different graphs [4].  The robustness can be assessed by measuring the detection probability of the mark and the bit error rate for a set of criteria that are relevant to the application, which is considered. Part of these evaluation profiles can be defined using a finite and precise set of robustness criteria (e.g., S.D.M.I., IFPI or E.B.U. requirements) and one just needs to check them.  False alarms are difficult to measure and we are working on a method to estimate them automatically without having to do an exhaustive search of the key space.  Finally, speed is very dependent on the type of implementation: software or hardware. Here we are only concerned with software implementation and our test just computes an average of the time required on a particular given platform to watermark and image depending on its size.